Voice signature for user authentication to electronic device

ABSTRACT

An authentication application receives an audio input, detects whether the audio input matches a known passphrase, and processes the audio input to determine whether the audio input is consistent with a known voice signature. Upon determining that the audio input is consistent with the known voice signature, the application will identify a user who is associated with the voice signature. The device will output an audio prompt, receive a spoken answer, and analyze the spoken answer to determine whether it corresponds to an expected response. Outputting the prompt may responsive to determining that the user was not authenticated to the device or the application within a threshold time period, or if a security threat was posted to the user&#39;s profile, When the system determines that the spoken answer corresponds to the expected response, it will authenticate the user and provide the user with access to the device or application.

RELATED APPLICATIONS AND CLAIM OF PRIORITY

This patent document claims priority to U.S. Provisional PatentApplication No. 62/458,146, filed Feb. 13, 2017, the disclosure of whichis fully incorporated into this document by reference.

BACKGROUND

Voice activation of electronic devices is growing in popularity.Currently, voice activation requires that each electronic device betrained to recognize the user's voice. Although a user's voice promptscould be stored on a remote server, there could be some delay inauthentication and extra data use caused by the remote processing. Inaddition, some users may prefer that remote storage of their recordedvoice not occur at all.

Although physical keys exist that can store biometric data and use thestored data to authenticate a user to an electronic device, physicalkeys can be misplaced or lost. In addition, there are limited ways touse physical keys to authenticate users on many types of electronicdevices.

In addition, certain electronic devices such as digital home assistantsmay be accessed by multiple users, some of whom may use the same orsimilar passphrases. When different users access the same device, it isdesirable to know which user is accessing the device so that the systemcan access the correct user's profile and provide a personalized usageexperience to the user.

This disclosure describes improved methods and systems for verifying theidentity of an account user by voice across multiple electronic devices.

SUMMARY

In various embodiments, a method of authenticating a user to anelectronic device includes, by a processor of the electronic device,executing an authentication application by: (a) causing a microphone ofthe electronic device to receive a spoken audio input; (b) analyzing thespoken audio input to detect whether the spoken audio input matches aknown passphrase; and (c) processing the spoken audio input to determinewhether the audio input is consistent with a known voice signature. Upondetermining that the spoken audio input is consistent with the knownvoice signature, the processor will identify a user who is associatedwith the known voice signature.

The system may determine whether the user was previously authenticatedto the electronic device, or to an application of the electronic device,within a threshold time period. Alternatively or in addition, the systemmay determine whether a security threat has been posted to the user'ssystem usage profile. If the processor determines that the user was notpreviously authenticated to the electronic device or the applicationwithin the threshold time period, or if an indication of a securitythreat has been posted to the user's profile, or upon detection of otherdetermined conditions, the system will: (i) cause the electronic deviceto output an audio prompt with an expected response; (ii) cause themicrophone to receive a spoken answer in response to the audio prompt;(iii) analyze the spoken answer to determine whether the spoken answercorresponds to the expected response; and (iv) after determining thatthe spoken answer corresponds to the expected response, authenticate theuser and provide the user with access to the electronic device or theapplication.

Before providing the user with access, in response to detecting that thespoken answer corresponds to the expected response, the processor mayprocess the audio input to determine whether the spoken answer isconsistent with the known voice signature. If so, it may only providethe user with the access if the spoken answer is consistent with theknown voice signature.

In some embodiments, processing the spoken audio input to determinewhether the spoken audio input is consistent with the known voicesignature may include identifying one or more user profiles that includethe spoken audio input, selecting each user that is associated with oneof the identified one or more user profiles as a candidate user, andcomparing the spoken audio input with voice signatures of each of thecandidate users to identify a candidate user whose voice signature isconsistent with the spoken audio input.

Optionally, before processing the spoken audio input to determinewhether the audio input is consistent with the known voice signature,the system may generate the known voice signature by executing a voicesignature generation application that: (a) requests the user to speak aknown text passage; (b) receives a spoken audio instance of the knowntext passage; (c) processes the spoken audio instance to generate theknown voice signature; (d) transmits the known voice signature to aremote server for storage; and (e) deletes or does not save any audiorecording of the spoken audio instance.

Optionally, before processing the spoken audio input to determinewhether the audio input is consistent with the known voice signature,the system may generate the known voice signature by executing a voicesignature generation application that: (a) outputs a query; (b) receivesa spoken response to the query; (c) processes the spoken response toyield the known voice signature; (d) transmits the known voice signatureto a remote server for storage; and (e) deletes or does not save anyaudio recording of the spoken response so that the spoken audio instanceis not saved on the electronic device (although it may be saved at theremote server).

Optionally, the electronic device that executes the authenticationapplication and the electronic device that executes the voice signaturegeneration application may be separate physical devices. Alternatively,they may be the same device or components of the same device.

Optionally, before analyzing the spoken answer to determine whether thespoken answer corresponds to the expected response, the system maygenerate a profile for the user by outputting queries, receiving repliesto the queries, processing the replies to identify text contained ineach of the replies, and saving to a remote server a profile comprisingthe identified text for each reply in association with the reply'scorresponding query. When causing the electronic device to output theaudio prompt, the system may randomly select one of the queries from theprofile. When analyzing the spoken answer to determine whether thespoken answer corresponds to the expected response, the system maydetermine whether the spoken answer corresponds to the identified textfor the reply that corresponds to the selected query.

To determine whether a user's system usage profile includes anindication of a security threat, the system may determine whether thesystem usage profile includes one or more of the following: (i) anattempt to simultaneously access an application on more than a thresholdnumber of devices at the same time; (b) an attempt to simultaneously usemultiple devices in different geographic locations at the same time; or(c) an attempt to access an application or service that is inconsistentwith a typical usage pattern for the user.

In other embodiments, an electronic device user authentication systemincludes an electronic device having a microphone, a speaker and amemory containing an authentication application. When executed, theauthentication application will cause a microphone of the electronicdevice to receive a spoken audio input, cause one or more processors toanalyze the audio input to detect whether the audio input matches aspoken passphrase, and cause one or more processors to process the audioinput to determine whether the audio input is consistent with a knownvoice signature. Upon determining that the audio input matches thespoken passphrase and is consistent with the known voice signature, thesystem will identify a user who is associated with the known voicesignature. The system may access a profile for the user to determinewhether the profile includes either (a) an indication of a securitythreat, or (b) an indication that the user was not previouslyauthenticated to the electronic device or to a selected application ofthe electronic device within a threshold time period. The speaker of theelectronic device will output an audio prompt with an expected response.Optionally, outputting the audio prompt is done in response todetermining that the system usage profile includes an indication of asecurity threat or an indication that the user was not previouslyauthenticated within the threshold time period. After the microphonereceives a spoken answer to the audio prompt, the system will analyzethe spoken answer to determine whether the spoken answer corresponds tothe expected response. If the system can determine that the spokenanswer corresponds to the expected response, the system willauthenticate the user so that the user may access one or more functionsof the electronic device.

Optionally, the portion of the system that processes the audio input todetermine whether the audio input is consistent with a known voicesignature may be one or more components of a server that is remote fromthe electronic device and in communication with the electronic device.If so, the known voice signature need not be stored on the electronicdevice.

The system also may include a voice signature generation applicationconfigured to cause the speaker of the electronic device to output aquery or a request that the user to speak a known text passage. Thesystem may receive a spoken response via the microphone, process thespoken audio instance to generate the known voice signature, andtransmit the known voice signature to a remote server for storage. Theelectronic device may then delete, or at least not save, any audiorecording of the spoken audio instance. Thus, the spoken audio instancewill not be saved on the electronic device.

In an alternate embodiment, a method of authenticating a user to anelectronic device includes determining that a microphone of anelectronic device has received a spoken audio input, and using thespoken audio input to identify a user. The method also includes causingthe electronic device to output an audio prompt that is associated withan expected response. When it is determined that the microphone hasreceived a spoken answer to the audio prompt, the system will analyzethe spoken answer to determine whether the spoken answer corresponds tothe expected response. After determining that the spoken answercorresponds to the expected response, the system will authenticate theuser and provide the user with access to the electronic device or anapplication of the electronic device.

Optionally, before causing the electronic device to output the audioprompt, the system may determine that the user was not previouslyauthenticated to the electronic device or to the application within athreshold time period.

Optionally, using the spoken audio input to identify the user mayinclude: (a) analyzing the spoken audio input to detect whether thespoken audio input matches a known passphrase, and identifying a userwho is associated with the known passphrase; or (b) processing thespoken audio input to determine whether the spoken audio input isconsistent with a known voice signature, and identifying a user who isassociated with the known voice signature.

Optionally, after identifying the user, the system may access a systemusage profile for the user to determine whether the system user profileincludes an indication of a security threat. If the system determinesthat the system user profile includes an indication of a securitythreat, then in response it may implement the step of causing theelectronic device to output the audio prompt. Accessing the system usageprofile may include determining whether the system usage profileincludes one or more of the following: (a) an attempt to simultaneouslyaccess an application on more than a threshold number of devices at thesame time; (b) an attempt to simultaneously use multiple devices indifferent geographic locations at the same time; or (c) an attempt toaccess an application or service that is inconsistent with a typicalusage pattern for the user.

Optionally, before providing the user with the access, and in responseto detecting that the spoken answer corresponds to the expectedresponse, the system may process the spoken answer to determine whetherthe spoken answer is consistent with a known voice signature. If so,then it may only provide the user with the access if the spoken answeris consistent with the known voice signature.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example system for verifying an account user toany number of electronic devices according to an embodiment.

FIG. 2 is a flow chart of an example method of authenticating a userwith a two-step voice authentication process.

FIG. 3 illustrates a process of generating a voice signature.

FIG. 4 illustrates how an authentication application may build a userprofile with second-level authentication questions and expectedresponses.

FIG. 5 depicts a block diagram of hardware that may be used to containor implement program instructions.

DETAILED DESCRIPTION

Terminology that is relevant to this disclosure includes:

In this document, the singular forms “a,” “an,” and “the” include pluralreferences unless the context clearly dictates otherwise. The term“comprising” means “including, but not limited to.” Similarly, the term“comprises” means “includes, and is not limited to.” Unless definedotherwise, all technical and scientific terms used in this document havethe same meanings as commonly understood by one of ordinary skill in theart.

An “electronic device” or a “computing device” refers to a device orsystem that includes a processor and memory. Each device may have itsown processor and/or memory, or the processor and/or memory may beshared with other devices as in a virtual machine or containerarrangement. The memory will contain or receive programming instructionsthat, when executed by the processor, cause the electronic device toperform one or more operations according to the programminginstructions. Examples of electronic devices include personal computers,servers, mainframes, virtual machines, containers, gaming systems,televisions, digital home assistants and mobile electronic devices suchas smartphones, fitness tracking devices, wearable virtual realitydevices, Internet-connected wearables such as smart watches and smarteyewear, personal digital assistants, cameras, tablet computers, laptopcomputers, media players and the like. Electronic devices also mayinclude appliances and other devices that can communicate in anInternet-of-things arrangement, such as smart thermostats,refrigerators, connected light bulbs and other devices. Electronicdevices also may include components of vehicles such as dashboardentertainment and navigation systems, as well as on-board vehiclediagnostic and operation systems. In a client-server arrangement, theclient device and the server are electronic devices, in which the servercontains instructions and/or data that the client device accesses viaone or more communications links in one or more communications networks.In a virtual machine arrangement, a server may be an electronic device,and each virtual machine or container may also be considered to be anelectronic device. In the discussion below, a client device, serverdevice, virtual machine or container may be referred to simply as a“device” for brevity. Additional elements that may be included inelectronic devices will be discussed below in the context of FIG. 5.

The terms “processor” and “processing device” refer to a hardwarecomponent of an electronic device that is configured to executeprogramming instructions. Except where specifically stated otherwise,the singular terms “processor” and “processing device” are intended toinclude both single-processing device embodiments and embodiments inwhich multiple processing devices together or collectively perform aprocess.

The terms “memory,” “memory device,” “data store,” “data storagefacility” and the like each refer to a non-transitory device on whichcomputer-readable data, programming instructions or both are stored.Except where specifically stated otherwise, the terms “memory,” “memorydevice,” “data store,” “data storage facility” and the like are intendedto include single device embodiments, embodiments in which multiplememory devices together or collectively store a set of data orinstructions, as well as individual sectors within such devices.

FIG. 1 illustrates an example system for verifying an account user tovarious electronic devices according to an embodiment. As illustrated byFIG. 1, any number of client electronic devices, such as a smartphone102, computing device 103 such as the tablet computing device shown,on-board vehicle devices 104 such as in-dashboard entertainment andnavigation systems, wearable electronic devices such as smart watch 105,and connected Internet of Things devices such as home voice assistant106, smart light bulb 107 and smart thermostat 108, communicate witheach other and/or a remote server 101 via a communication network 110.The communication network 110 may be a local area network (LAN), a widearea network (WAN), a mobile or cellular communication network, anextranet, an intranet, the Internet and/or the like.

Each of the client electronic devices 102-108 will include a microphoneby which the device may receive audio inputs, and in particular voiceprompts from a device user. One or more of the client electronic devicesand/or the server 101 may include or be in communication with one ormore data stores 112. Each data store may include one or morenon-transitory computer-readable media sectors in which data or otherinformation can be stored.

In various embodiments, a user can use his or her voice as anauthentication key to access an electronic device or one or moreapplications that the electronic device executes. To do this, the systemgenerates a voice signature for the user by prompting the user torespond to a set of prompts on a single electronic device, and also tospeak a passphrase into the electronic device. The voice signature is aunique code set that includes data about certain characteristics of theuser's voice, but it does not include actual recordings of the user'svoice itself. In this document, the terms “voice print” and “voicesignature” are used interchangeably to refer to a voice signature. Thevoice signature and passphrase may be stored in a memory of theelectronic device. Or, to avoid potential copying of and access to thevoice signature and passphrase, either or both of these may be uploadedand stored to a data store that is remote from the electronic device ina cloud service.

Later, when the user wishes to access an application or electronicdevice, to verify the user's identity, the device listens (or theapplication causes the device to listen) for a unique passphrase to bespoken. When the passphrase is detected, the received audio signal isprocessed to identify certain characteristics of the audio signal, andthose characteristics are cross-referenced with the user's voicesignature.

If authentication is confirmed, then the electronic device may permitthe user to access it or the requested application. However, if the userhas not previously accessed the particular application or device, or ifthe user has not accessed the particular device within a thresholdperiod of time, or if a potential security threat has been associatedwith the user's system usage profile (such as because a security issuewas detected on the user's account since the user was lastauthenticated), then before granting the permission the authenticationprocess will perform a second verification step and ask the user tospeak a response to a question. The system will then parse the user'sspoken response and compare the response to the user's voice signature.If the response is consistent with the user's voice signature, thesystem will then authenticate the user to the device. Alternatively, thesecond verification step (or an additional verification step) may be touse a near field communication (NFC) receiver of the device to detectwhether another device that is known to be associated with the user iswithin a detection range of the NFC receiver, by determining whether itcan receive a signal from the associated device.

An example process flow for authenticating a user to an electronicdevice is shown in FIG. 2. In various embodiments, a processor of anelectronic device will execute an authentication application, which is aset of programming instructions that include a rule set and/orparameters for determining whether a user is authorized to access anelectronic device or one or more applications that are available to theuser on the electronic device. The method will include causing amicrophone of the electronic device to receive a spoken audio input 201.Optionally, before causing the microphone to receive the spoken audioinput, the system may output a visual or audio prompt to ask the user toprovide the spoken audio input 201, or the system may receive a physicalcommand from the user, such as a touch of a touchscreen activation area,a power button and/or a voice activation switch. Alternatively, thesystem may simply maintain its audio speaker in an “always on” mode sothat it continuously listens for the passphrase without first promptingthe user or receiving a physical command from the user.

The device may require that the user speak a known passphrase before itwill authenticate the user to the device or its application. If so, thenthe system may analyze the audio input to detect whether the audio inputmatches a known passphrase 202. The system may do this by any suitablespeech recognition process, such as by using a speech-to-text converterand comparing the resulting word or phrase to a data set of knownpassphrases that have been previously used to access the electronicdevice or the application. To authenticate the user to the device, thesystem may require that (among other criteria) the spoken passphrasematch one of the known passphrases. The known passphrases may be storedlocally on the device, or the known passphrases may be stored remotelyon a server that is running an authentication application. Knownpassphrases may be stored in combination with user profiles, so that thesystem can identify a candidate user (or set of candidate users) as afirst level of analysis 203. Candidate users are those users having auser profile that includes or is associated with the spoken passphrase.

If the audio input does not match a known passphrase, the device willcontinue monitoring the environment via the microphone until it detectsa known passphrase 204.

When the system detects that an audio input matches a known passphrase,the system will also process the audio input to generate a voice printfor the audio input 205. The system will then determine whether thevoice print for the audio input is consistent with a known voicesignature that is stored in a user profile 206. The system may generatevoice prints and determine whether a spoken phrase is consistent with avoice print. Methods of generating voice prints and determiningconsistency are known to those of skill in the art, such as thosedisclosed in U.S. Pat. Nos. 9,595,260 and 9,398,128, the disclosures ofwhich are incorporated into this document by reference. The embodimentsof this disclosure may use any now or hereafter known voice printgeneration and matching process. The system may initially compare thevoice print to the voice signatures of those users who were identifiedas candidate users in step 203. If there are no matches in the set ofcandidate users, the system also may compare the voice print to voicesignatures of one or more other users for whom the system has userprofiles that contain voice signatures. The system may select the otherusers using any suitable criteria, such as users who are known priorusers of the electronic device.

If the system cannot identify any user having a voice signature thatmatches the audio input's voice print, then the system may deny 220 theuser access to the application or device until the user authenticateshimself or herself to the device, such as by physically entering orspeaking a username and password, or by presenting a biometricidentifier such as a fingerprint or retina scan to a scanner of theelectronic device, or by presenting a physical token.

On the other hand, if the system determines that the audio input isconsistent with a known voice signature 206, then the system mayidentify the user whose profile includes the known voice signature. Thesystem may also extract data from the user profile or another memorysegment of a data store and use that data to determine, based on theuser's usage history that is stored in the profile or other memorysegment, whether the user last performed one or more qualifying accessevents within a threshold time period 207. A qualifying access event mayinclude using the same electronic device (optionally with the sameapplication or a different application), or using the same applicationon a different known electronic device), or some other action. If theuser whose profile includes the known voice signature performed aqualifying access event within the threshold time period, the system maygrant the user access to the application or the electronic device 217and thus permit the user to access the device or application. If thesystem cannot identify a voice signature that corresponds to a knownuser, or if there is a match but the user did not perform a qualifyingaccess event within the threshold time period, the system may requirethe user to perform a second level authentication process, which will bedescribed below.

Optionally, the system may implement one or more additional rules thatestablish criteria for deciding whether to grant the user access orrequire the user to perform the second level authentication process. Forexample, the authentication application may extract data from the user'ssystem usage profile to determine whether the system usage profileindicates that a potential security threat has been posted to the user'ssystem usage profile or otherwise occurred within a threshold timeperiod 208. Examples of security issues that can result in a securitythreat being posted include detection of an attempt to access anapplication on more than a threshold number of devices at the same time,detection of an attempt to use multiple devices in different geographiclocations at the same time, or detection of an attempt to access anapplication or service that is inconsistent with the user's typicalusage pattern (such as detecting an access attempt in a country wherethe user had not previously done so). The system usage profile may be acomponent of the user profile described above, or it may be a separateprofile that is stored in a separate memory segment or memory device.

If the system determines that the user should undergo a second levelauthentication process before granting the user access, the system mayselect an audio prompt and cause a speaker of the electronic device tooutput the audio prompt to the user 210. The prompt will be a question,statement or phrase that is associated with an expected response, andwill be stored in the user's profile or be based on data that iscontained in the user's profile. For example, the prompt may be aquestion such as “what is your favorite movie” or “what was your firstpet's name,” and the response may be stored in the user's profile. Thequestion may be randomly selected from a set of candidate questions, orquestions may be selected from a candidate set using any suitablecriteria, such as selecting a question that was not previously askedwithin a threshold period of time or in a threshold number of mostrecent previous questions. The question may require the user to performa spoken task, such as to speak an additional passphrase or securitycode.

The system may receive, via a microphone of the electronic device, aspoken response to the prompt 211. The authentication application willthen process the spoken response to determine whether it matches theexpected response 212, using any suitable speech recognition technologysuch as those previously described. In some embodiments, the system mayoutput multiple prompts, and the user may be required to select one ormore of them, or select all of them.

If the spoken response for each presented prompt matches the expectedresponse, the system may authenticate the user and permit the user toaccess the device or application 217. If the spoken response does notmatch the expected response, the authentication application may deny 220the user access to the application or device until the user providesadditional authentication credentials. Alternatively, the authenticationapplication may select and output one or more different questions untileither the user provides a correct expected response (in which case theauthentication application will grant access) or a threshold number ofquestions have been presented and not correctly answered (in which casethe authentication application will deny access).

Optionally, the system also may generate a voice print of the user'sanswer to each prompt in the second level authentication process, andthe system may compare the voice print to the voice signature that iscontained in the user's profile 213. If this happens, the authenticationapplication may also require that the voice print of the response matchthe voice signature before it will grant the user access 214. Otherwise,it may deny 220 the user access to the application or device until theuser provides an additional valid authentication credential.

Before implementing the processes described in FIG. 2, the system maygenerate and store the voice signature, as well as the prompts and theirassociated expected responses. FIG. 3 describes processes that thesystem may follow to do this. Referring to FIG. 3, a processor of anelectronic device, executing a voice signature generation applicationthat outputs a prompt that requests the user to speak a known textpassage 301. The electronic device may be the same device or a differentdevice as those used in the authentication processes of FIG. 2. The textpassage will be retrieved from a memory that is accessible by the voicesignature generation application, or contained in a rule set of thevoice signature generation application. The voice signature generationapplication may output this text passage by displaying it on a displayscreen of an electronic device, by causing a speaker of an electronicdevice to output it as spoken words, by causing a printer to print it,by sending it to the user's messaging system address in an electronicmessage, or by some other means. The system will then receive a spokenaudio instance of the known text passage 302 and process the spokenaudio instance 303 using now or hereafter known processes such as thosedescribed above to generate the known voice signature 304. Theelectronic device will then transmit the known voice signature to aremote server for storage so that other electronic devices can access itfor their authentication processes 305. The voice signature may be savedas part of, or in association with, a user profile for the user. Thesystem will not transmit any audio recording of the user's speaking ofthe text passage to the remote server. In some embodiments, even theelectronic device may delete the spoken response after processing iscomplete so that an actual audio recording of the user speaking isarchived.

Optionally, before analyzing the spoken audio input the system may alsogenerate or supplement the user profile to obtain a set of prompts andexpected responses. This may be done using any suitable method. Forexample, referring to FIG. 4, the authentication application may causean electronic device user interface 401 to output a list, drop-downmenu, or other structure containing candidate questions. The user mayselect questions from the candidate set and provide answers that will besaved and used as expected answers. The system will save the text ofeach answer to the user's profile. Alternatively, the system maygenerate questions and expected answers based on data that is stored inthe user's system usage profile, such as “when did you last use thisapplication?” or “what was the last song you listened to on thiselectronic device′?” As another alternative, the system may provide theuser an ability to enter questions and answers as freeform text orspoken words.

FIG. 5 depicts a block diagram of hardware that may be used to containor implement program instructions, such as those of a hosted service, amonitoring service for a hosted service, an electronic device that isaccessing a hosted service, or a virtual machine or container thatserves in any of these roles. A bus 500 serves as an information highwayinterconnecting the other illustrated components of the hardware. Thebus may be a physical connection between elements of the system, or awired or wireless communication system via which various elements of thesystem share data. Processor 505 is a processing device of the systemperforming calculations and logic operations required to execute aprogram. Processor 505, alone or in conjunction with one or more of theother elements disclosed in FIG. 5, is an example of a processingdevice, computing device or processor as such terms are used within thisdisclosure. The processing device may be a physical processing device, avirtual device contained within another processing device, or acontainer included within a processing device.

A memory device 510 is a hardware element or segment of a hardwareelement on which programming instructions, data, or both may be stored.Read only memory (ROM) and random access memory (RAM) constituteexamples of memory devices, along with cloud storage services.

An optional display interface 530 may permit information to be displayedon the display 535 in audio, visual, graphic or alphanumeric format.Communication with external devices, such as a printing device, mayoccur using various communication devices 540, such as a communicationport or antenna. A communication device 540 may be communicativelyconnected to a communication network, such as the Internet or anintranet.

The hardware may also include a user input such as a keyboard or keypad550, or other input device such as a mouse, a touch pad, a touch screen,a remote control, a pointing device, or a video input device. Thehardware also may include a microphone 555 for receipt of voice andother audio inputs. Data also may be received from an image capturingdevice 520 such as a digital camera or video camera. A positional sensor560 and/or motion sensor 570 may be included to detect position andmovement of the device. Examples of motion sensors 570 includegyroscopes or accelerometers. Examples of positional sensors 560 such asa global positioning system (GPS) sensor device that receives positionaldata from an external GPS network.

The processes described above can help electronic devices authenticate auser to an account without requiring the user to enter a password or usea physical key. An alternative use could be to unlock or activatephysical security to a building or vehicle. It could also be used toverify someone's identity over the phone when dealing with medical orfinancial information. Multiple passkeys could be stored in the datastore so that a randomly chosen challenge question can be solved by auser as the second level authentication. The use of voice recognitioncan also enable multiple users to be authenticated to a single device,so long as voice prints for each user are generated and stored by theauthentication system.

The features and functions described above, as well as alternatives, maybe combined into many other different systems or applications. Variousalternatives, modifications, variations or improvements may be made bythose skilled in the art, each of which is also intended to beencompassed by the disclosed embodiments.

The invention claimed is:
 1. A method of authenticating a user to anelectronic device, the method comprising: by a processor of theelectronic device, executing an authentication application by: causing amicrophone of the electronic device to receive a spoken audio input;analyzing the spoken audio input to detect whether the spoken audioinput matches a known passphrase, wherein the known passphrase isincluded in a plurality of user profiles, wherein each user profile isassociated with a respective user of a plurality of candidate users;upon detecting that the spoken audio input matches the known passphraseincluded in the plurality of user profiles, comparing the spoken audioinput with voice signatures in each of the plurality of user profiles;identifying, based on the comparing of the spoken audio input with thevoice signatures, a user among the plurality of candidate users with avoice signature in the user's profile consistent with the spoken audioinput; determining whether the user performed one or more qualifyingaccess events within a threshold time period of receipt of the spokenaudio input; implementing one or more rules to decide between (i)providing the user with access to the electronic device based on thedetermination that the user performed the one or more qualifying accessevents within the threshold time period and (ii) requiring the user toundergo a second level authentication process; in response todetermining that the user should undergo the second level authenticationprocess, performing the second level authentication process by: causingthe electronic device to select a query among a plurality of queries ina profile of the user, wherein the query is associated with an expectedresponse and has not been previously asked within a threshold period oftime, causing the electronic device to output an audio prompt based onthe selected query, causing the microphone to receive a spoken answer tothe audio prompt, analyzing the spoken answer to determine whether thespoken answer corresponds to the expected response, and after decidingto provide the user with access or determining that the spoken answercorresponds to the expected response, authenticating the user andproviding the user with access to the electronic device.
 2. The methodof claim 1 wherein: implementing the rules to decide whether to requirethe user to undergo the second level authentication process comprisesdetermining whether the user was previously authenticated to theelectronic device, or to a selected application of the electronicdevice, within a threshold time period; and wherein determining that theuser should undergo the second level authentication process comprisesdetermining that the user was not previously authenticated to theelectronic device or the selected application within the threshold timeperiod.
 3. The method of claim 1 wherein: implementing the rules todecide whether to require the user to undergo the second levelauthentication process comprises accessing a system usage profile forthe user to determine whether the system usage profile includes anindication of a security threat; and wherein causing the electronicdevice to output the audio prompt is performed in response todetermining that the system usage profile includes an indication of asecurity threat.
 4. The method of claim 3, wherein accessing the systemusage profile for the user to determine whether the system usage profileincludes an indication of a security threat comprises determiningwhether the system usage profile includes one or more of the following:an attempt to simultaneously access an application on more than athreshold number of devices at the same time; an attempt tosimultaneously use multiple devices in different geographic locations atthe same time; or an attempt to access an application or service that isinconsistent with a typical usage pattern for the user.
 5. The method ofclaim 1 further comprising, before providing the user with the access:in response to detecting that the spoken answer corresponds to theexpected response, processing the spoken answer to determine whether thespoken answer is consistent with the voice signature in the user'sprofile; and only providing the user with the access if the spokenanswer is consistent with the voice signature in the user's profile. 6.The method of claim 1 further comprising, before identifying the user,generating the voice signature in the user's profile by: executing avoice signature generation application that: requests the user to speaka known text passage, receives a spoken audio instance of the known textpassage, processes the spoken audio instance to generate the voicesignature, transmits the voice signature to a remote server for storage,and deletes or does not save any audio recording of the spoken audioinstance.
 7. The method of claim 6, wherein the electronic device thatexecutes the authentication application and the electronic device thatexecutes the voice signature generation application are separatephysical devices.
 8. The method of claim 1 further comprising, beforeidentifying the user, generating the voice signature in the user'sprofile by: executing a voice signature generation application that:outputs a query, receives a spoken response to the query, processes thespoken response to yield the voice signature, transmits the voicesignature to a remote server for storage, and deletes or does not saveany audio recording of the spoken response so that the spoken responseis not saved on the electronic device.
 9. The method of claim 1, furthercomprising: generating the profile for the user by: outputting theplurality of queries, receiving a plurality of audio replies to thequeries, processing the replies to identify text contained in each ofthe audio replies, and saving, to a remote server, the profilecomprising the identified text for each reply in association with thereply's corresponding query; and when analyzing the spoken answer todetermine whether the spoken answer corresponds to the expectedresponse, determining whether the spoken answer corresponds to theidentified text for the reply that corresponds to the selected query.10. An electronic device user authentication system, comprising: anelectronic device comprising a microphone, a speaker and a memorycontaining an authentication application that, when executed by one ormore processors, is configured to: cause the microphone to receive anaudio input; cause the one or more processors to analyze the audio inputto detect whether the audio input matches a known passphrase, whereinthe known passphrase is included in a plurality of user profiles,wherein each user profile is associated with a respective user of aplurality of candidate users; cause the one or more processors to, upondetecting that the spoken audio input matches the known passphraseincluded in the plurality of user profiles, compare the spoken audioinput with voice signatures in each of the plurality of user profiles;cause the one or more processors to identify, based on the comparing ofthe spoken audio input with the voice signatures, a user among theplurality of candidate users with a voice signature in the user'sprofile consistent with the spoken audio input; determine whether theuser performed one or more qualifying access events within a thresholdtime period of receipt of the audio input; implement one or more rulesto decide between (i) providing the user with access to the electronicdevice based on the determination that the user performed the one ormore qualifying access events within the threshold time period and (ii)requiring the user to undergo a second level authentication process; inresponse to determining that the user should undergo the second levelauthentication process, perform the second level authentication processby: causing the electronic device to select a query among a plurality ofqueries in a profile of the user, wherein the query is associated withan expected response and has not been previously asked within athreshold period of time, causing the speaker of the electronic deviceto output an audio prompt based on the selected query, and after themicrophone receives a spoken answer in response to the audio prompt,causing the one or more processors to analyze the spoken answer todetermine whether the spoken answer corresponds to the expectedresponse; and after deciding to provide the user with access ordetermining that the spoken answer corresponds to the expected response,cause the one or more processors to authenticate the user so that theuser may access one or more functions of the electronic device.
 11. Theelectronic device user authentication system of claim 10, wherein: theone or more rules to decide whether to require the user to undergo thesecond level authentication process comprise rules to access a profilefor the user to determine whether the profile includes an indicationthat the user was not previously authenticated to the electronic deviceor to a selected application of the electronic device within a thresholdtime period; and determining that the user should undergo the secondlevel authentication process comprises determining that the profileincludes an indication that the user was not previously authenticatedwithin the threshold time period.
 12. The electronic device userauthentication system of claim 10, wherein: the one or more rules todecide whether to require the user to undergo the second levelauthentication process comprise rules to access a profile for the userto determine whether the profile includes an indication of a securitythreat; and determining that the user should undergo the second levelauthentication process comprises determining that the profile includesan indication of a security threat.
 13. The electronic device userauthentication system of claim 10, wherein: the one or more processorsare one or more components of a server that is remote from theelectronic device and in communication with the electronic device; andthe voice signatures are not stored on the electronic device.
 14. Theelectronic device user authentication system of claim 10, furthercomprising additional instructions that are configured to cause the oneor more processors to, before providing the user with the access: inresponse to detecting that the spoken answer corresponds to the expectedresponse, process the audio input to determine whether the spoken answeris consistent with the voice signature in the user's profile; and onlyprovide the user with the access if the spoken answer is consistent withthe voice signature in the user's profile.
 15. The electronic deviceuser authentication system of claim 10, further comprising a voicesignature generation application configured to cause the electronicdevice to: cause the speaker of the electronic device to output a queryor a request that the user to speak a known text passage; receive, viathe microphone, a spoken response; process the spoken response togenerate the voice signature; transmit the voice signature to a remoteserver for storage, and delete or not save any audio recording of thespoken response so that the spoken response is not saved on theelectronic device.
 16. A method of authenticating a user to anelectronic device, the method comprising: by a processor: determiningthat a microphone of an electronic device has received a spoken audioinput; determining that the spoken audio input matches a knownpassphrase, wherein the known passphrase is included in a plurality ofuser profiles, wherein each user profile is associated with a respectiveuser of a plurality of candidate users, upon determining that the spokenaudio input matches the known passphrase included in the plurality ofuser profiles, comparing the spoken audio input with voice signatures ineach of the plurality of user profiles; identifying, based on thecomparing of the spoken audio input with the voice signatures, a useramong the plurality of candidate users with a voice signature in theuser's profile consistent with the spoken audio input; determiningwhether the user performed one or more qualifying access events within athreshold time period of receipt of the spoken audio input; implementingone or more rules to decide between (i) providing the user with accessto the electronic device based on the determination that the userperformed the one or more qualifying access events within the thresholdtime period and (ii) requiring the user to undergo a second levelauthentication process; in response to determining that the user shouldundergo the second level authentication process, performing the secondlevel authentication process by: causing the electronic device to selecta query among a plurality of queries in a profile of the user, whereinthe query is associated with an expected response and has not beenpreviously asked within a threshold period of time, causing theelectronic device to output an audio prompt based on the selected query,determining that the microphone has received a spoken answer to theaudio prompt, and analyzing the spoken answer to determine whether thespoken answer corresponds to the expected response; and after decidingto provide the user with access or determining that the spoken answercorresponds to the expected response, authenticating the user andproviding the user with access to the electronic device or anapplication of the electronic device.
 17. The method of claim 16,wherein: implementing the one or more rules to decide whether to requirethe user to undergo the second level authentication process comprisesdetermining that the user has not been previously authenticated to theelectronic device or to the application within a threshold time period.18. The method of claim 16, wherein implementing the one or more rulesto decide whether to require the user to undergo the second levelauthentication process comprises: accessing a system usage profile forthe user to determine whether the system usage profile includes anindication of a security threat; and determining that the system usageprofile includes an indication of a security threat.
 19. The method ofclaim 18, wherein accessing the system usage profile for the user todetermine whether the system usage profile includes an indication of asecurity threat comprises determining whether the system usage profileincludes one or more of the following: an attempt to simultaneouslyaccess an application on more than a threshold number of devices at thesame time; an attempt to simultaneously use multiple devices indifferent geographic locations at the same time; or an attempt to accessan application or service that is inconsistent with a typical usagepattern for the user.
 20. The method of claim 16 further comprising,before providing the user with the access: in response to detecting thatthe spoken answer corresponds to the expected response, processing thespoken answer to determine whether the spoken answer is consistent withthe voice signature in the user's profile; and only providing the userwith the access if the spoken answer is consistent with the voicesignature in the user's profile.